- 4782 Views
- 7 replies
- 4 kudos
Resolved! Databricks Asset Bundles complex variable for cluster configuration substitute
Using this page of the DAB docs, I tried to substitute cluster configuration by a variable. That way, I want to predefine different job cluster configurations. Doing exactly what is used in the docs yields this error:Error: failed to load [...]/speci...
- 4782 Views
- 7 replies
- 4 kudos
- 797 Views
- 1 replies
- 1 kudos
Logging in Databricks
Hi All,I am trying to create a log using python logging package.Is this allowed in databricks and any sample working code that you can share?Thank you for your guidance.Best regards,NC
- 797 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @NC ,I recommend below video that guides you how to use logging in databricks:Introduction to Logging and Quality Control in Databricks (youtube.com)
- 1 kudos
- 947 Views
- 1 replies
- 1 kudos
Seeking Advice on Data Lakehouse Architecture with Databricks
I'm currently designing a data lakehouse architecture using Databricks and have a few questions. What are the best practices for efficiently ingesting both batch and streaming data into Delta Lake? Any recommended tools or approaches?
- 947 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @joshbuttler,I think the best way is to use auto loader, which provides a highly efficient way to incrementally process new data, while also guaranteeing each file is processed exactly once.It supports ingestion in a batch mode (Trigger.Available...
- 1 kudos
- 1702 Views
- 1 replies
- 1 kudos
Best Practices as a Beginner
Hello everyone,I am working on a project where I need to conduct some analysis on a dataset with 1 billion rows. I extracted the Parquet file from Azure and saved it onto the DBFS. Every time I want to run SQL queries and do preprocessing/analysis, I...
- 1702 Views
- 1 replies
- 1 kudos
- 1 kudos
@Faiyaz17 wrote:Hello everyone,I am working on a project where I need to conduct some analysis on a dataset with 1 billion rows. I extracted the Parquet file from Azure and saved it onto the DBFS. Every time I want to run SQL queries and do preproces...
- 1 kudos
- 28342 Views
- 9 replies
- 9 kudos
Resolved! How to specify entry_point for python_wheel_task?
Can someone provide me an example for a python_wheel_task and what the entry_point field should be?The jobs UI help popup says this about "entry_point":"Function to call when starting the wheel, for example: main. If the entry point does not exist in...
- 28342 Views
- 9 replies
- 9 kudos
- 9 kudos
Just want to confirm - my project uses PDM not poetryand as such uses[project.entry-points.packages]Rather than[tool.poetry.scripts]and the bundle is failing to run on the cluster - as it can't find the entry point - is this expected behavior?
- 9 kudos
- 2127 Views
- 1 replies
- 2 kudos
Resolved! DLT create_table vs create_streaming_table
What is the difference between the create_table and create_streaming_table functions in dlt?For example, this is how I have created a table that streams data from kafka written as json files to a volume. @Dlt.table( name="raw_orders", table_...
- 2127 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @ggsmith ,If you check the examples, you will notice that dlt.create_streaming_table is more specialized and you may consider it to be your target.As per documentation:Check this example:https://www.reddit.com/r/databricks/comments/1b9jg3t/dedupin...
- 2 kudos
- 429 Views
- 0 replies
- 0 kudos
[Filter: SPAM] filipniziol's post body matched "sugar", board "data-engineering".
[Filter: SPAM] filipniziol's post body matched "sugar", board "data-engineering". Post Subject: Re: DLT create_table vs create_streaming_table Post Body: Hi @ggsmith , If you check the examples, you will notice that dlt.create_streaming_table is ...
- 429 Views
- 0 replies
- 0 kudos
- 13329 Views
- 5 replies
- 0 kudos
Need to automatically rerun the failed jobs in databricks
Hi all, I need to retrigger the failed jobs automatically in data bricks, can you please help me with all the possible ways to make it possible
- 13329 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @Vishalakshi ,I have responded during the weekend, but it seems the responses were lost.You have here the run object. For example the current criteria is to return only runs where run[state][result_state] == "FAILED" so basically all failed jobs.W...
- 0 kudos
- 1591 Views
- 2 replies
- 1 kudos
Permission Issue with IDENTIFIER clause
Hi all,we are parameterizing environment specific catalog names (like `mycatalog_dev` vs. `mycatalog_prd`) in Lakeview dashboard queries like this:SELECT * FROM IDENTIFIER(:catalog_name || '.myschema.mytable')Which works fine in most cases. We have o...
- 1591 Views
- 2 replies
- 1 kudos
- 1 kudos
I've had quite a bit of fun with UC and view permissions. I don't think this is specific to using the IDENTIFIER() function, but I suspect it's related to UC permissions. What you'll need to ensure:The user or group who owns the view on catalog_b h...
- 1 kudos
- 4874 Views
- 2 replies
- 2 kudos
File Trigger using azure file share in unity Catalog
Hello, I have got the unity catalog eanbled in my workspace. the file srae manually copied by customers in azure file share(domain joint account, wabs) on adhoc basis. I would like to add a file trigger on the job so that as soon as file arrives in t...
- 4874 Views
- 2 replies
- 2 kudos
- 2 kudos
@Diego33 Kaniz is half-bot half-human, but unfortunately not gracing us with "sorry for the confusion" responses.After a quick search, I thought that maybe there's a possiblity to find use the web terminal and do a manual mount with the bash script t...
- 2 kudos
- 2187 Views
- 2 replies
- 3 kudos
DBUtils from databricks-connect and runtime are quite different libraries....
If you find yourself using dbutils in any of your code, and you're testing locally vs running on a cluster, there's a few gotchas to be very careful of when it comes to listing files in Volumes or files on DBFS.The DBUtils you'll use locally installe...
- 2187 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @stevenayers-bge ,Thanks for sharing. I didn't know that these interfaces aren't align with each other.
- 3 kudos
- 1791 Views
- 2 replies
- 0 kudos
Resolved! Cannot install jar to cluster: invalid authority.
Hi allI want to access on-prem Oracle Database data from the python notebooks. However, the install of the jar (ojdbc8.jar) results in an error, which occurs while the cluster is starting up.The error message:"Library installation attempted on the dr...
- 1791 Views
- 2 replies
- 0 kudos
- 0 kudos
The error message suggests that the jar file located at abfss:/jars/ojdbc8.jar has an invalid authority. This could be due to a number of reasons such as incorrect file path, insufficient permissions, or network restrictions. Here are a few steps you...
- 0 kudos
- 2991 Views
- 6 replies
- 9 kudos
Problem with dropDuplicates in Databricks runtime 15.4LTS
Hi,I'm testing the latest version of the databricks runtime but I'm getting errors doing a simple dropDuplicates.Using the following codedata = spark.read.table("some_table") data.dropDuplicates(subset=['SOME_COLUMN']).count() I'm getting this error....
- 2991 Views
- 6 replies
- 9 kudos
- 9 kudos
Unless is was communicated as a breaking changes between major updates, it would be OK. But I can't find anything in the release notes, so it's a bug.
- 9 kudos
- 3630 Views
- 3 replies
- 2 kudos
Reading data from S3 in Azure Databricks
Is it possible to create an external volume in Azure Databricks that points to an external S3 bucket so that I can read files for processing? Or is it only limited to ADLSv2?
- 3630 Views
- 3 replies
- 2 kudos
- 2 kudos
Yep, I'm keen to see this functionality as well.I think it is reasonable to expect external locations can be on diverse storage types (at least the big players). I can nicely control access to azure storage in UC but not S3.
- 2 kudos
- 21981 Views
- 8 replies
- 3 kudos
Resolved! Is it possible to passthrough job's parameters to variable?
Scenario:I tried to run notebook_primary as a job with same parameters' map. This notebook is orchestrator for notebooks_sec_1, notebooks_sec_2, and notebooks_sec_3 and next. I run them by dbutils.notebook.run(path, timeout, arguments) function.So ho...
- 21981 Views
- 8 replies
- 3 kudos
- 3 kudos
I am using getCurrentBindings(), but it returns an empty dictionary even though I passed parameters. I am running it in a scheduled workflow job
- 3 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 485 | |
| 349 | |
| 287 |
