- 6659 Views
- 1 replies
- 1 kudos
Resolved! how can I pass the df columns as a parameter
Im doing the self study and want pass df column name as a parameter.I have defined the widget column_name= dbutils.widgets.get('column_name')which is executing succefuly ( giving me a column name)then Im reading the df and do some transformation and ...
- 6659 Views
- 1 replies
- 1 kudos
- 1 kudos
df2.select([column_name]).writeORdf2.select(column_name).write
- 1 kudos
- 31767 Views
- 2 replies
- 6 kudos
Resolved! Difference between "spark.table" & "spark.read.table"?
Hi,I want to make a PySpark DataFrame from a Table. I would like to ask about the difference of the following commands:spark.read.table(TableName)&spark.table(TableName)Both return PySpark DataFrame and look similar. Thanks.
- 31767 Views
- 2 replies
- 6 kudos
- 6 kudos
Hi @Kaniz Fatma​ I selected answer from @Kedar Deshpande​ as the best answer.
- 6 kudos
- 3776 Views
- 2 replies
- 0 kudos
Faced error using Databricks SQL Connector
I installed databricks-sql-connector in Pycharm.Then i run the query below based on docs.I refer this docs.(https://docs.databricks.com/dev-tools/python-sql-connector.html)==========================================from databricks import sqlimport osw...
- 3776 Views
- 2 replies
- 0 kudos
- 0 kudos
It seems that one of your environment variables is incorrect. Please print them and compare them with the connection settings from the cluster or SQL warehouse endpoint.
- 0 kudos
- 50902 Views
- 6 replies
- 11 kudos
Resolved! how to find the size of a table in python or sql?
let's suppose there is a database db, inside that so many tables are there and , i want to get the size of tables . how to get in either sql, python, pyspark.even if i have to get one by one it's fine.
- 50902 Views
- 6 replies
- 11 kudos
- 11 kudos
@Raman Gupta​ - could you please try the below %python spark.sql("describe detail delta-table-name").select("sizeInBytes").collect()
- 11 kudos
- 9542 Views
- 1 replies
- 6 kudos
How can I simplify my data ingestion by processing the data as it arrives in cloud storage?
This post will help you simplify your data ingestion by utilizing Auto Loader, Delta Optimized Writes, Delta Write Jobs, and Delta Live Tables. Pre-Req: You are using JSON data and Delta Writes commandsStep 1: Simplify ingestion with Auto Loader Delt...
- 9542 Views
- 1 replies
- 6 kudos
- 6 kudos
This post will help you simplify your data ingestion by utilizing Auto Loader, Delta Optimized Writes, Delta Write Jobs, and Delta Live Tables.Pre-Req: You are using JSON data and Delta Writes commandsStep 1: Simplify ingestion with Auto Loader Delta...
- 6 kudos
- 3294 Views
- 0 replies
- 1 kudos
How can i save a parquet file using pandas with a data factory orchestrated notebook?
Hi guys,this is my first question, feel free to correct me if i'm doing something wrong.Anyway, i'm facing a really strange problem, i have a notebook in which i'm performing some pandas analysis, after that i save the resulting dataframe in a parque...
- 3294 Views
- 0 replies
- 1 kudos
- 2045 Views
- 0 replies
- 4 kudos
Default location for Schema/Database in Unity
Hello Bricksters,We organize the delta lake in multiple storage accounts. One storage account per data domain and one container per database. This helps us to isolate the resources and cost on the business domain level.Earlier, when a schema/database...
- 2045 Views
- 0 replies
- 4 kudos
- 2138 Views
- 1 replies
- 3 kudos
Cluster list in Microsoft.Azure.Databricks.Client fails because ClusterSource enum does not include MODELS. When you have a model serving cluster, Clu...
Cluster list in Microsoft.Azure.Databricks.Client fails because ClusterSource enum does not include MODELS.When you have a model serving cluster, ClustersApiClient.List method fails to deserialize the API response because that cluster has MODELS as C...
- 2138 Views
- 1 replies
- 3 kudos
- 4932 Views
- 3 replies
- 4 kudos
Unity through terraform
I am working on automation of Unity through terraform. I have referred below link link to get started :https://registry.terraform.io/providers/databricks/databricks/latest/docs/guides/unity-catalog-azureI am facing issue when I create metastore using...
- 4932 Views
- 3 replies
- 4 kudos
- 4 kudos
Not sure if you got this working, but I noticed you are using provider: `databrickslabs/databricks`, hence why this is not avaialable. You should be using new provider: `databricks/databricks`: https://registry.terraform.io/providers/databricks/datab...
- 4 kudos
- 2349 Views
- 1 replies
- 1 kudos
Resolved! How to get started with Auto Loader using partner academy portal? Are there any videos and step by step material
Need Video and step by step documentation on Auto Loader as well as how to build end-to-end data pipeline
- 2349 Views
- 1 replies
- 1 kudos
- 1 kudos
@raja iqbal​ below course will provide overview related Autoloader Course name: How to Use Databricks' Auto Loader for Incremental ETL with the Databricks Data Science and Data Engineering WorkspaceIf you register for Data Engineer Catalog, then you ...
- 1 kudos
- 15878 Views
- 7 replies
- 10 kudos
Is there any way to propagate errors from dbutils?
I have a master notebook that runs a few different notebooks on a schedule using the dbutils.notebook.run() function. Occasionally, these child notebooks will fail (due to API connections or whatever). My issue is, when I attempt to catch the errors ...
- 15878 Views
- 7 replies
- 10 kudos
- 10 kudos
I have the same issue. I see no reason that Databricks couldn't propagate the internal exception back through their WorkflowException
- 10 kudos
- 5542 Views
- 1 replies
- 2 kudos
AnalysisException: Multiple sources found for bigquery (com.google.cloud.spark.bigquery.BigQueryRelationProvider, com.google.cloud.spark.bigquery.v2.BigQueryTableProvider), please specify the fully qualified class name.
While reading data from BigQuery to Databricks getting the error : AnalysisException: Multiple sources found for bigquery (com.google.cloud.spark.bigquery.BigQueryRelationProvider, com.google.cloud.spark.bigquery.v2.BigQueryTableProvider), please spe...
- 5542 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Parul Paul​ , could you please check if this is the scenario: https://stackoverflow.com/questions/68623803/load-to-bigquery-via-spark-job-fails-with-an-exception-for-multiple-sources-foun Also, you can refer: https://github.com/GoogleCloudDatapro...
- 2 kudos
- 3396 Views
- 4 replies
- 1 kudos
Bug in db.fs.utils
When using db.fs.utils on a s3 bucket titled "${sometext}.${sometext}.${somenumber}${sometext}-${sometext}-${sometext}" we receive an error. PLEASE understand this is an issue with how it encodes the .${somenumber} because we verified with boto3 that...
- 3396 Views
- 4 replies
- 1 kudos
- 1 kudos
@Debayan Mukherjee​ All the information is there please read accurately. I am not going to give you the actual bucket name I am using on a public forum. As i said above here is the command:dbutils.fs.ls("s3a://${bucket_name_here_follow_above_format}"...
- 1 kudos
- 13578 Views
- 3 replies
- 6 kudos
Resolved! AnalysisException: The schema of your Delta table has changed in an incompatible way since your DataFrame or DeltaTable object was created. Please redefine your DataFrame or DeltaTable object.
can anybody tell me why this error is coming and what's the reliable solution for it
- 13578 Views
- 3 replies
- 6 kudos
- 6 kudos
Hi @Raman Gupta​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...
- 6 kudos
- 4487 Views
- 2 replies
- 7 kudos
- 4487 Views
- 2 replies
- 7 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 554 | |
| 349 | |
| 287 |
