- 3570 Views
- 3 replies
- 1 kudos
databricks multistage jobs in workflow Params not passing
I am using a multi-stage job calling different notebooks all have the same PARAMNAME that needs to be passed in. one the second and third job, I input the new a different PARAM's value .. but those values do not show up when it runs the task. I...
- 3570 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @David Byrd​ this is already a known thing and we have raised it to our engineering team. If you have the same key but different values in the parameters, then its most likely takes the first value for the key and will use the same for all the tas...
- 1 kudos
- 5289 Views
- 7 replies
- 1 kudos
Handling Changing Schema in CDC DLT
We are building a DLT pipeline and the autoloader is handling schema evolution fine. However, further down the pipeline we are trying to load that streamed data with the apply_changes() function into a new table and, from the looks of it, doesn't see...
- 5289 Views
- 7 replies
- 1 kudos
- 1 kudos
Hey there @Palani Thangaraj​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear fro...
- 1 kudos
- 6457 Views
- 2 replies
- 8 kudos
Resolved! AWS and Azure on the same Databricks workspace.
Can I use compute instances from different providers i.e. AWS and Azure on the same Databricks workspace?
- 6457 Views
- 2 replies
- 8 kudos
- 8 kudos
Hi @Liam ODonoghue​ there are a few methods using which you can connect AWS and Azure resources. In such cases, it involves only your accounts. But with Databricks, you need to handle two accounts for both cloud providers. Let's say if you create a w...
- 8 kudos
- 1579 Views
- 2 replies
- 2 kudos
Terraform managing workspaces
Hi , Do we have any references? I am actually looking for managing workspaces and other databricks related stuff with terraform, trying to create in the form of modules so that everytime i need to create a new workspace i just need to call the worksp...
- 1579 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi there @Rayan D​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...
- 2 kudos
- 2327 Views
- 2 replies
- 0 kudos
Why is my Plotly graph squished?
fig = make_subplots(1,4) cols = ['OrderValue', 'TransactionPrice', 'ProductPrice', 'ProductUnits'] for i, col in enumerate(cols): fig.add_trace( go.Histogram(x=silver_df.select(col).toPandas()[col]), row=1, col=i+1 ) p =...
- 2327 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey there @Niels Ota​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.T...
- 0 kudos
- 3100 Views
- 2 replies
- 2 kudos
Databricks jobs UPdate/Reset API throws an unexpected error
{ "error_code": "INVALID_PARAMETER_VALUE", "message": "Missing required field: job_id"}I have a test job cluster and I need to update the docker image filed with the other version using reset/update job API. I went through the documentation of data b...
- 3100 Views
- 2 replies
- 2 kudos
- 2 kudos
Hey there @radha kilaru​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from yo...
- 2 kudos
- 3758 Views
- 2 replies
- 0 kudos
Unable to execute Data Engineering Scripts in Community Edition
I am getting error while executing scripts 6.1 onward with various error messages. All the earlier scripts were fine but not any more. I am not sure what is the problem and whom to contact. Multiple images are attached below from examples 6.1 and 6.2...
- 3758 Views
- 2 replies
- 0 kudos
- 5795 Views
- 3 replies
- 1 kudos
Resolved! Error: "Backend service unavailable" when trying to start a cluster in community edition.
I am using community edition of databricks for learning and hands-on projects. However, when I try to create a cluster today, I am getting an error popup- "Backend service unavailable". I would like to know if it is a problem with my account or a bac...
- 5795 Views
- 3 replies
- 1 kudos
- 1 kudos
Hey there @Venkat K​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...
- 1 kudos
- 5237 Views
- 3 replies
- 1 kudos
Resolved! Can I add custom cluster tag from init script ?
Hi, is it possible to add custom tags from init script during cluster initialization ? We would like to automatically add custom tags whenever someone creates a new cluster in databricks.
- 5237 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Md Tahseen Anam​ I don't think there is a possibility to use an init script for cust tags. But the easiest way is to use cluster policies. You can mention a list of custom tags in the policy so that you can simply add the policy to the cluster wh...
- 1 kudos
- 5464 Views
- 3 replies
- 2 kudos
Resolved! How to read multiple tiny XML files in parallel
Hi team, we are trying to read multiple tiny XML files, able to parse them using the data bricks XML jar, but is there any way to read these files in parallel and distribute the load across the cluster? right now our job is taking 90% of the time rea...
- 5464 Views
- 3 replies
- 2 kudos
- 2 kudos
Thank you @Hubert Dudek​ for the suggestion. Similar to your recommendation, we added a step in our pipeline to merge the small files to large files and make them available for the spark job.
- 2 kudos
- 36528 Views
- 1 replies
- 0 kudos
Resolved! Exporting data from databricks to external csv
I need to export some data from the database to csv which will be downloaded to another application. What would be the procedure for that? I don't have a lot of knowledge in DataBricks and I didn't find much information in the documentation.Thanks.
- 36528 Views
- 1 replies
- 0 kudos
- 0 kudos
You can manually download data to your local in CSV from databricks notebook cell and pass it to your another application.Your application can run Databricks notebook inside a workflow via an API that writes data to S3 bucket in CSV and in response y...
- 0 kudos
- 2090 Views
- 2 replies
- 2 kudos
- 2090 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @rajat kumar​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...
- 2 kudos
- 2537 Views
- 2 replies
- 0 kudos
Azure SA mounted but can't load files
I am attempting to load an excel file that's located in a blob storage that I've mounted. In the first cell, when I use the dbutils.fs.ls command, I can see the file I want to load. However, when I try to actually load it, it can't find the file. It ...
- 2537 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Niels Ota​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 0 kudos
- 3093 Views
- 3 replies
- 0 kudos
Is it possible to install databricks-cli on a shared laptop.
I have a government furnished laptop (GFE). My normal user is not an admin on the laptop. I have a separate admin login on the laptop. I was able to install databricks-cli as the admin user but it installed under that users appdata python path which ...
- 3093 Views
- 3 replies
- 0 kudos
- 0 kudos
Hey there @John Zajic​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....
- 0 kudos
- 7112 Views
- 3 replies
- 4 kudos
Resolved! DeltaTable' object has no attribute 'clone'
Hello, I use delta on my local machine and I would like to clone a table, however the cloning is not working.I have the last version of delta installed (delta-spark==2.0.0) but the clone method does not exist in the python module.With this code :delt...
- 7112 Views
- 3 replies
- 4 kudos
- 4 kudos
Hello @Bertrand BURCKER​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from yo...
- 4 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
