- 3951 Views
- 2 replies
- 2 kudos
Resolved! Access Azure storage with serverless compute
I would like to know how to connect to Azure Blob Storage in a Python job inside a workflow with serverless cluster. When working with a non-serverless cluster or with serverless in a declarative pipeline, I would typically set the Azure storage acco...
- 3951 Views
- 2 replies
- 2 kudos
- 2 kudos
Use the below code in your notebook. You cannot set spark config in serverless as there is no advanced options in cluster.credential_id = dbutils.secrets.get(scope="{scope_name}",key="{app_id}") credential_key = dbutils.secrets.get(scope="{scope_name...
- 2 kudos
- 2214 Views
- 2 replies
- 2 kudos
how to edit or delete a post in this community after posted?
when trying to edit the post i am not able see the edit option there. @Retired_mod
- 2214 Views
- 2 replies
- 2 kudos
- 2 kudos
It’s actually pretty simple, just click the three dots on the top right of your post, and you’ll see options to edit or delete. Easy to miss at first, but once you see it, it’s straightforward.
- 2 kudos
- 1783 Views
- 5 replies
- 0 kudos
How to import Class Room Setup Scripts -03.4
I am learning Databricks Data engineering certificate and during the course i try to load the class room scripts in my databricks community edition and i am getting following error message. I am relatively new to databricks
- 1783 Views
- 5 replies
- 0 kudos
- 0 kudos
@nayan_wylde , szymon_dybczakEven if i can complete the few courses then it should be fine for me. please share step by step guide how to import these libraries in personal workspace and run notebooks.
- 0 kudos
- 68856 Views
- 8 replies
- 11 kudos
Resolved! Automate the Databricks workflow deployment
Hi everyone,Do you guys know if it's possible to automate the Databricks workflow deployment through azure devops (like what we do with the deployment of notebooks)?
- 68856 Views
- 8 replies
- 11 kudos
- 11 kudos
Did you get a chance to try Brickflows - https://github.com/Nike-Inc/brickflowYou can find the documentation here - https://engineering.nike.com/brickflow/v0.11.2/Brickflow uses - Databricks Asset Bundles(DAB) under the hood but provides a Pythonic w...
- 11 kudos
- 854 Views
- 1 replies
- 0 kudos
org.apache.spark.SparkException: Job aborted due to stage failure: org.apache.spark.memory.SparkOutO
Hi everyone,I'm using a serverless cluster and encountering an issue where my code runs fine when executed cell-by-cell in a notebook, but fails with a memory error when executed as a job. Interestingly, the same job runs successfully in our dev envi...
- 854 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Parth2692! It’s possible that your dev and prod environments have different serverless configurations, which could explain the difference in behavior. You can try increasing the notebook memory by switching from Standard to High in the Environ...
- 0 kudos
- 3428 Views
- 1 replies
- 1 kudos
Autoloader with availableNow=True and overwrite mode removes data in second micro-batch (DBR 16.3)
Hi everyone,I'm encountering an issue after upgrading to Databricks Runtime 16.3, while using Autoloader with the following configuration:trigger(availableNow=True)outputMode("overwrite")When a new file arrives, Autoloader processes it and writes the...
- 3428 Views
- 1 replies
- 1 kudos
- 1 kudos
You've hit on a known behavioral change or subtle interaction in Databricks Runtime 16.3 with Autoloader, trigger(availableNow=True), and outputMode("overwrite"). This specific combination seems to be causing an unexpected second micro-batch that ove...
- 1 kudos
- 4715 Views
- 3 replies
- 3 kudos
Resolved! How to use bundle substitutions in %pip install for Lakeflow Declarative Pipelines
Hello,When defining a Lakeflow Declarative Pipeline (DLT pipeline) I would like to allow the installation of a whl file to be dictated by the user running the pipeline. This will allow the notebook to have the pip installs at the top be agnostic of t...
- 4715 Views
- 3 replies
- 3 kudos
- 3 kudos
Glad to help and feel free to "Accept as Solution" to help others in the same boat Cheers, Lou.
- 3 kudos
- 1407 Views
- 2 replies
- 2 kudos
Resolved! How to deploy a DLT Pipeline?
Hi community,My team and I have been working on manually creating our first DLT pipeline. However, when we tried importing it into DABs, we encountered an issue in the dev workspace: we are unable to deploy the same DLT pipeline multiple times becaus...
- 1407 Views
- 2 replies
- 2 kudos
- 2 kudos
Hello,Thanks for your response! Duplicating the catalog for this does feel a bit unusual. I understand the reasoning behind it, though it’s not the cleanest approach. Still, I suppose it’s acceptable for a DEV workspace.Thanks again!
- 2 kudos
- 1421 Views
- 1 replies
- 1 kudos
Notebook execution keeps showing "Fetching result" endlessly
Hello, I am executing a very simple notebook with only two cells. In the first cell, I'm just defining some variables and printing the result. The second cell is more complex and it basically grabs those variables, parses a yaml file, and prints the ...
- 1421 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @himoshi Good Day!1. Could you use print() instead of display()?2.If you're printing a large YAML or dictionary object directly (e.g. print(parsed_yaml) or display(parsed_yaml)),try:import jsonprint(json.dumps(parsed_yaml, indent=2)[:500]) # Print...
- 1 kudos
- 1443 Views
- 3 replies
- 2 kudos
Where is the path settings for New SQL Editory?
Hi everyone,I have switched the "New SQL Editor" on SQL Editor side. When opened this new feature, every new query saves automatically under my user. This is crazy, I cannot manage my folders like this. But I didn't find any article or message about ...
- 1443 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @lingareddy_Alva ,Many thanks for your response. Currently I'm using the "Save As" method that you mentioned above. But I'm looking for the default path settings. Because, in my method, I'm using a lot of Editor querying tables. I know that's not ...
- 2 kudos
- 1639 Views
- 1 replies
- 0 kudos
Accessing DBT Articafts, runs, tests from Databricks workflow using automated script.
I am running dbt on a databricks job. It saves all documentation: manifest.json, run_results.json, etc in "Download Artifacts" in a job. I am not able to find out a way to read those in codes, transform and save on databricks. Tried job API. The arti...
- 1639 Views
- 1 replies
- 0 kudos
- 0 kudos
I know it is a late thread, but did you solve this? I am running into the same challange you have. It seems you can use get/job, and expand the field "tasks" to get the task ids. and then you can use the get-output from that task id. If that helpshtt...
- 0 kudos
- 5816 Views
- 6 replies
- 5 kudos
Issue with HTML Table Styling in Databricks Alerts
Hi Community,I’m trying to create an alert in Databricks with a custom email notification that includes the results of a SQL query displayed in an HTML table. However, I am facing issues with styling the table, specifically with adding borders and ba...
- 5816 Views
- 6 replies
- 5 kudos
- 3598 Views
- 3 replies
- 1 kudos
lab mismatch with the course
Hi I am taking a lab included course on "Data Ingestion with Lakeflow Connect" and the labs shown in the course doesn't match with the lab opened in vocareum. The title of the vocareum lab does match the course title but demo and labs are different. ...
- 3598 Views
- 3 replies
- 1 kudos
- 1 kudos
@Kishori, could you please file a support ticket? The team will be able to review the course details and assist you directly.You can raise a ticket here: https://help.databricks.com/s/contact-us?ReqType=training
- 1 kudos
- 1141 Views
- 2 replies
- 0 kudos
Not able to read sample data in databricks in shared cluster but using single cluster im able to
Im not able to view sample data using share clusterError getting sample datasocket closedBut when I use the single cluster mode Im able to read the data
- 1141 Views
- 2 replies
- 0 kudos
- 0 kudos
I've enabled the outbound traffic on port 8443 but still its not working please help me out here
- 0 kudos
- 1548 Views
- 2 replies
- 0 kudos
Newest version of dbx-workspace always returns NoneType
I just updated the `databricks-sdk` library to the newest version on PyPi, and for some reason, I am almost always getting this error:File "/home/site/wwwroot/.python_packages/lib/site-packages/databricks/sdk/credentials_provider.py", line 283, in to...
- 1548 Views
- 2 replies
- 0 kudos
- 0 kudos
It seems your issue is in getting the AAD token. You are using an SPN to authenticate. You can try to update the azure packages too.azure-identity>=1.21.0azure-core>=1.32.0azure-mgmt-core>=1.6.0databricks-sdk>=0.57.0
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 566 | |
| 349 | |
| 287 |
