I want to use Databricks to analyze data that is stored on an on-premises SQL Server. What is the best way to bring this data into Databricks?I tried to configure Lakehouse Federation so I could query on-premises data directly from Databricks. Howeve...
So far, I started 2 Databricks Academy Courses. In both the course, the default is hive-metastore where the schema is created. In my org, hive metastore is blocked and we have been asked to use Unity Catalog. Is there a way the course material in dat...
I have workflow like this.1 task: job as a task. Inside this job there is task which is seting parameter x as taskValue using dbutils.jobs.taskValues.set. 2. task dependent on previous job as a task. I would like to access this parameter x. I tried t...
I see, I have requested for someone else to guide you on this. cc: @Retired_mod
Hi - I have existing parquet files in Cloudflare R2 storage (created outside of Databricks). I would like to share them via Delta Share, but I keep running into an error. Is it possible to share existing parquet files without duplicating them?I did...
Thanks @Retired_mod. It's currently possible to share a delta table stored in an S3 external location without duplication or doing the `DEEP CLONE` first. Is it on the roadmap to support this for R2 as well?
Good Evening, I am configuring databricks_mws_credentials through Terraform on AWS. I am getting the following error:Error: cannot create mws credentials: invalid Databricks Account configuration││ with module.databricks.databricks_mws_credentials.t...
Hello, I'm facing a similaire Issue. I try to update my TF with properly authentification and I have this error ?! ╷ │ Error: cannot create mws credentials: failed visitor: context canceled │ │ with databricks_mws_credentials.this, │ on main.tf ...
I am asking this because up to now I have just seen some examples of deploying a pre-existent kedro project in databricks in order to run some pipelines...
Hi! Kedro Dev here. You can surely develop Kedro on Databricks, in fact we have a lot of Kedro project running on Databricks. In the past there has been some friction, mainly because Kedro are project based while Databricks focus a lot on notebook. T...
I am trying to migrate legacy dashboards from source workspace to target workspace using python code. Is there any API's to GET legacy dashboards and POST API for migrating. When I am trying to migrate legacy dashboards and the underlying queries, I ...
Hello,I can't seem to find up to date info on how to handle catalog.schema.table in DLT live table and Unity catalog.My statement is the following and is failing with the error: Multipart table name is not supported. Any workaround possible?Thanks a ...
Hi @Pierre1 ,Actually, you don't provide this infromation in the code. You specify this information when you create dlt pipeline.If you do not select a catalog and target schema for a pipeline, tables are not published to Unity Catalog and can only b...
I work with Azure Databricks. The shortcut Ctrl + / to toggle line comment doesn't work on AZERTY keyboard on Firefox... Do you know this issue ? Is there an other shortcut I can try ? Thanks !
'cmd + shift + 7' works for me!I'm using an AZERTY keyboard on Chrome for MacOS.
Is it possible to migrate Databricks Legacy Dashboards along with associated components such as queries and datasets from one workspace to another? I have attempted to export the Legacy Dashboards along with the queries and datasets, but upon importi...
Currently I am working with a project where we need to mask PIIs in few columns for VIP customers only.Let me explain briefly with example:Table A: [personid, status, address, UID, VIPFLAG] --> Mask "UID" and "address" only where VIPFLAG is 1Table ...
For example here I extract the schedule parameter as a complex type variable: variables: schedule: description: schedule time type: complex default: quartz_cron_expression: '0 22 17 * * ?' timezone_id: Asia/Shanghai pa...
If the validation is fine on your colleague's laptop and not on yours, my first assumption would be that it's a version issue. Do you have the same Databricks CLI version as your colleagues? You can check with databricks --version Also according to...
Hi Folks,I tested running a merge update using SQL queries in a notebook, and it is faster than using a Scala script. Both tests were done using the same cluster size in Databricks.How can I make the Scala script as fast as the SQL notebook?
Have you already compared both query plans?
I would like to be able to get custom error's message ideally visible from Workflows > Jobs UI.1. For example, workflow failed because file was missing and could not find it, in this case I am getting "Status" Failed and "Error Code" RunExecutionErro...
What you can do is pass the custom error message you want from the notebook back to the workflow output = f"There was an error with {error_code} : {error_msg}"dbutils.notebook.exit(output) Then when you are fetching the status of your pipeline, you c...
i want to write all my part file into a single text file is there anything i can do
When writing a pyspark dataframe to a file, it will always write to a part file by default. This is because of partitions, even if there is only 1 partitions.To write into a single file you can convert the pyspark dataframe to a pandas dataframe and ...
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 485 | |
| 349 | |
| 287 |
