- 12864 Views
- 6 replies
- 0 kudos
Resolved! Error: cannot create mws credentials: invalid Databricks Account configuration
Good Evening, I am configuring databricks_mws_credentials through Terraform on AWS. I am getting the following error:Error: cannot create mws credentials: invalid Databricks Account configuration││ with module.databricks.databricks_mws_credentials.t...
- 12864 Views
- 6 replies
- 0 kudos
- 0 kudos
Hello, I'm facing a similaire Issue. I try to update my TF with properly authentification and I have this error ?! ╷ │ Error: cannot create mws credentials: failed visitor: context canceled │ │ with databricks_mws_credentials.this, │ on main.tf ...
- 0 kudos
- 61622 Views
- 3 replies
- 0 kudos
Just a doubt, can we develop a kedro project in databricks?
I am asking this because up to now I have just seen some examples of deploying a pre-existent kedro project in databricks in order to run some pipelines...
- 61622 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi! Kedro Dev here. You can surely develop Kedro on Databricks, in fact we have a lot of Kedro project running on Databricks. In the past there has been some friction, mainly because Kedro are project based while Databricks focus a lot on notebook. T...
- 0 kudos
- 685 Views
- 0 replies
- 0 kudos
Migrating legacy dashboards from source workspace to target workspace
I am trying to migrate legacy dashboards from source workspace to target workspace using python code. Is there any API's to GET legacy dashboards and POST API for migrating. When I am trying to migrate legacy dashboards and the underlying queries, I ...
- 685 Views
- 0 replies
- 0 kudos
- 3865 Views
- 1 replies
- 2 kudos
DLT with Unity Catalog: Multipart table name
Hello,I can't seem to find up to date info on how to handle catalog.schema.table in DLT live table and Unity catalog.My statement is the following and is failing with the error: Multipart table name is not supported. Any workaround possible?Thanks a ...
- 3865 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Pierre1 ,Actually, you don't provide this infromation in the code. You specify this information when you create dlt pipeline.If you do not select a catalog and target schema for a pipeline, tables are not published to Unity Catalog and can only b...
- 2 kudos
- 51278 Views
- 6 replies
- 8 kudos
Resolved! Toggle line comment
I work with Azure Databricks. The shortcut Ctrl + / to toggle line comment doesn't work on AZERTY keyboard on Firefox... Do you know this issue ? Is there an other shortcut I can try ? Thanks !
- 51278 Views
- 6 replies
- 8 kudos
- 8 kudos
'cmd + shift + 7' works for me!I'm using an AZERTY keyboard on Chrome for MacOS.
- 8 kudos
- 1009 Views
- 0 replies
- 0 kudos
How to migrate Legacy Dashboard from one workspace to another workspace?
Is it possible to migrate Databricks Legacy Dashboards along with associated components such as queries and datasets from one workspace to another? I have attempted to export the Legacy Dashboards along with the queries and datasets, but upon importi...
- 1009 Views
- 0 replies
- 0 kudos
- 1143 Views
- 0 replies
- 1 kudos
Integrating row and column level security in parent child tables with masking only selected rows
Currently I am working with a project where we need to mask PIIs in few columns for VIP customers only.Let me explain briefly with example:Table A: [personid, status, address, UID, VIPFLAG] --> Mask "UID" and "address" only where VIPFLAG is 1Table ...
- 1143 Views
- 0 replies
- 1 kudos
- 2901 Views
- 3 replies
- 1 kudos
Resolved! Complex type variable in Databricks.yml not working
For example here I extract the schedule parameter as a complex type variable: variables: schedule: description: schedule time type: complex default: quartz_cron_expression: '0 22 17 * * ?' timezone_id: Asia/Shanghai pa...
- 2901 Views
- 3 replies
- 1 kudos
- 1 kudos
If the validation is fine on your colleague's laptop and not on yours, my first assumption would be that it's a version issue. Do you have the same Databricks CLI version as your colleagues? You can check with databricks --version Also according to...
- 1 kudos
- 771 Views
- 1 replies
- 1 kudos
Merge Update in Notebook Faster Than Scala script
Hi Folks,I tested running a merge update using SQL queries in a notebook, and it is faster than using a Scala script. Both tests were done using the same cluster size in Databricks.How can I make the Scala script as fast as the SQL notebook?
- 771 Views
- 1 replies
- 1 kudos
- 1 kudos
Have you already compared both query plans?
- 1 kudos
- 5610 Views
- 3 replies
- 2 kudos
Resolved! Error Handling and Custom Messages in Workflows
I would like to be able to get custom error's message ideally visible from Workflows > Jobs UI.1. For example, workflow failed because file was missing and could not find it, in this case I am getting "Status" Failed and "Error Code" RunExecutionErro...
- 5610 Views
- 3 replies
- 2 kudos
- 2 kudos
What you can do is pass the custom error message you want from the notebook back to the workflow output = f"There was an error with {error_code} : {error_msg}"dbutils.notebook.exit(output) Then when you are fetching the status of your pipeline, you c...
- 2 kudos
- 2546 Views
- 2 replies
- 0 kudos
Writing part files in single text file
i want to write all my part file into a single text file is there anything i can do
- 2546 Views
- 2 replies
- 0 kudos
- 0 kudos
When writing a pyspark dataframe to a file, it will always write to a part file by default. This is because of partitions, even if there is only 1 partitions.To write into a single file you can convert the pyspark dataframe to a pandas dataframe and ...
- 0 kudos
- 5731 Views
- 4 replies
- 4 kudos
Resolved! Get the list of loaded files from Autoloader
Hello,We can use Autoloader to track the files that have been loaded from S3 bucket or not. My question about Autoloader: is there a way to read the Autoloader database to get the list of files that have been loaded?I can easily do this in AWS Glue j...
- 5731 Views
- 4 replies
- 4 kudos
- 4 kudos
@Herry Ramli​ - Would you be happy to mark Hubert's answer as best so that other members can find the solution more easily?Thanks!
- 4 kudos
- 755 Views
- 0 replies
- 0 kudos
Dlt pipleline with unity catalog and external tables
We were using dlt pipeline with our raw and enhanced layer ( on hive metastore) but recently upgraded to unity catalog. We have external tables(storing data on different s3 bucket and metadata for table at unity catalog).At the moment dlt doest suppo...
- 755 Views
- 0 replies
- 0 kudos
- 12906 Views
- 6 replies
- 0 kudos
"Unable to upload to DBFS Query" Error running SQL warehouse query?
I have sql warehouse endpoints that work fine when querying from applications such as Tableau, but just running the included sample query against a running endpoint from the Query Editor from the workspace is returning "Unable to upload to DBFS Query...
- 12906 Views
- 6 replies
- 0 kudos
- 0 kudos
Hi @Marvin Ginns​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
- 0 kudos
- 2151 Views
- 1 replies
- 1 kudos
For dummies: How to avoid 'bill shock' & control AWS charges while learning to use Databricks?
Hi, I'm an out-of-work data analyst wanting to re-skill as a 'citizen data engineer'. By following how-to guides I was able to set up my own Databricks account etc along with a personal VPC in AWS. After 2 weeks of problem-free training I checked my...
- 2151 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi Hantha Databricks needs VPCs to work, but there are the default ones and customer managed ones: https://docs.databricks.com/en/security/network/classic/customer-managed-vpc.htmlCustomer managed ones are optional, but many tutorials include them as...
- 1 kudos
| User | Count |
|---|---|
| 1637 | |
| 791 | |
| 534 | |
| 349 | |
| 287 |
