- 8100 Views
- 4 replies
- 0 kudos
BINARY data type
Hello everyone.I'm trying to understand how BINARY data type works in spark sql. According to examples in the documentation, using cast or literal 'X' should return HEX representation of the binary data type, but when I try the same code, I see base6...
- 8100 Views
- 4 replies
- 0 kudos
- 0 kudos
If you are confused , please look at this thread, they explain that Databricks use base64 as binary default. This is not documented but can be tracked at source code level.https://stackoverflow.com/questions/75753311/not-getting-binary-value-in-datab...
- 0 kudos
- 1235 Views
- 1 replies
- 0 kudos
Shutdown Cluster in script if there is any failure
I am working on incremental load from sql server to Delta lake tables stored in ADLS gen2. DUring the script i need to qrite a logic toShut down the DB cluster on failure (there needs to be logging added to ensure that shutdown happens promptly to pr...
- 1235 Views
- 1 replies
- 0 kudos
- 0 kudos
If you run your notebook via workflow and error happen and there are no retires on job, then job cluster will be terminated immidietly after failure.You can add python block of try catch and if error occurs , you catch the error and log somewhere bef...
- 0 kudos
- 1462 Views
- 1 replies
- 0 kudos
Remove description from job
How do I remove a description from a job completely? When I try to just remove the text in the edit window, the same text shows up afterwards, even though it says "Successfully updated job". Also I had to write this twice, because on the first try I ...
- 1462 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi,this is not possible form UI You have to replace content with e.g. white space. I think this is bug.But you can do it using job api !Below example in PowerShell, just reaplce:job_istokenworkspaceURL$body = @' { "job_id": 123456789, "new_setti...
- 0 kudos
- 6788 Views
- 5 replies
- 5 kudos
How to change cluster size using a script
I want to change instance type or number of max workers via a python script. Does anyone know how to do it/is it possible? I have a lot of background jobs when I want to scale down my workers, so autoscaling is not an option. I was getting an error t...
- 6788 Views
- 5 replies
- 5 kudos
- 5 kudos
Hi ksenija, this is just my guess but maybe you are using Cluster Policy in your cluster that only allows you to use specific cluster size ? E.g. below cluster policy that limits to some cluster sizes only.
- 5 kudos
- 6561 Views
- 6 replies
- 3 kudos
Change DLT table type from streaming to 'normal'
I have a DLT streaming live table, and after watching a QA session, I saw that it is advised to only use streaming tables for your raw landing. I attempted to modify my pipeline to have my silver table be a regular LIVE TABLE, but an error was throw...
- 6561 Views
- 6 replies
- 3 kudos
- 3 kudos
Just curious, could you point me to said QA session if it's a video or something? I'm not aware of such a limitation. You can use DLT's live streaming tables anywhere in the Medallion architecture, just make sure not to break stream composability by ...
- 3 kudos
- 9059 Views
- 2 replies
- 0 kudos
Delta Lake, CFD & SCD2
HiWhat's the best way to deal with SCD2-styled tables in silver and/or gold layer while streaming.From what I've seen in the Professional Data Engineer videos, they usually go for SCD1 tables (simple updates or deletes)In a SCD2 scenario, we need to ...
- 9059 Views
- 2 replies
- 0 kudos
- 0 kudos
I did some further reading and got the same conclusion. APPLY CHANGES might to the trick. However, I don't like the limitations. From Bronze to Silver I might need .foreachBatch to implement the JSON-logic and the attribute names (__start_at / __end_...
- 0 kudos
- 1714 Views
- 1 replies
- 0 kudos
Resource exhaustion when using default apply_changes python functionality
Hello!We are currently setting up streaming CDC pipelines for more than 500 tables. Due to the high number of tables, we split our tables into multiple pipelines, we use multiple DLT pipelines per layer: bronze, silver goldIn silver, we only upsert ...
- 1714 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Lena1,there is no magic behind the scene.If you write readstream from bronze table and writestream with ForEachBatch(function) and in function you will write MERGE stattemnt this will have similiar performance.Maybe there is a lot of shuffeling ha...
- 0 kudos
- 8173 Views
- 5 replies
- 1 kudos
Resolved! My jobs and cluster were deleted in a suspicious way
I want to know what happen with my cluster and if I can recover it.I entered to my Databricks account and I didn't found my jobs and my cluster. I couldn't find any log of the deleted cluster because the log is into the cluster interface. I entered t...
- 8173 Views
- 5 replies
- 1 kudos
- 1 kudos
Dear folks,When the tables has been deleted, then why I am unable to create the table with same name.It continiously giving me error"DeltaAnalysisException: Cannot create table ('`spark_catalog`.`default`.`Customer_Data`'). The associated location ('...
- 1 kudos
- 12499 Views
- 6 replies
- 3 kudos
TesseractNotFoundError
TesseractNotFoundError: tesseract is not installed or it's not in your PATH. See README file for more information. in databricks
- 12499 Views
- 6 replies
- 3 kudos
- 3 kudos
%sh apt-get install -y tesseract-ocr this command is not working in my new Databricks free trail account, earlier it worked fine in my old Databricks instance. I get below error: E: Could not open lock file /var/lib/dpkg/lock-frontend - open (13: Per...
- 3 kudos
- 35895 Views
- 4 replies
- 0 kudos
How do you read files from the DBFS with OS and Pandas Python libraries?
I created translations for decoded values and want to save the dictionary object the DBFS for mapping. However, I am unable to access the DBFS without using dbutils or PySpark library. Is there a way to access the DBFS with OS and Pandas Python libra...
- 35895 Views
- 4 replies
- 0 kudos
- 0 kudos
db_path = 'file:///Workspace/Users/l<xxxxx>@databricks.com/TITANIC_DEMO/tested.csv' df = spark.read.csv(db_path, header = "True", inferSchema="True")
- 0 kudos
- 7312 Views
- 3 replies
- 0 kudos
Resolved! Delta live tables target schema
The first time i run my delta live table pipeline after setup, I get this error on starting it :-------------------------------------org.apache.spark.sql.catalyst.parser.ParseException: Possibly unquoted identifier my-schema-name detected. Please con...
- 7312 Views
- 3 replies
- 0 kudos
- 0 kudos
This still errors on internal databricks spark/python code likedeltaTable.history()@shagun wrote:The first time i run my delta live table pipeline after setup, I get this error on starting it :-------------------------------------org.apache.spark.sql...
- 0 kudos
- 5390 Views
- 1 replies
- 2 kudos
Databricks asset bundles use cluster depending on target (environment) is possible?
Here is my bundle definition Spoiler# This is a Databricks asset bundle definition for my_project.# See https://docs.databricks.com/dev-tools/bundles/index.html for documentation.experimental: python_wheel_wrapper: truebundle: name: my_projectinc...
- 5390 Views
- 1 replies
- 2 kudos
- 6561 Views
- 2 replies
- 1 kudos
Why is Spark creating multiple jobs for one action?
I noticed that when launching this bunch of code with only one action, I have three jobs that are launched.from pyspark.sql import DataFrame from pyspark.sql.types import StructType, StructField, StringType from pyspark.sql.functions import avgdata:...
- 6561 Views
- 2 replies
- 1 kudos
- 1 kudos
The above code will create two jobs.JOB-1. dataframe: DataFrame = spark.createDataFrame(data=data,schema=schema)The createDataFrame function is responsible for inferring the schema from the provided data or using the specified schema.Depending on the...
- 1 kudos
- 836 Views
- 0 replies
- 0 kudos
NonDeterministic UDF's making multiple invocations
Hi,Even after defining my UDF's as nonDeterministic like in here ..testUDF = udf(testMthod, StringType()).asNondeterministic()is still making multiple invocations. Is there any thing that I am missing here ?TIA,-John B
- 836 Views
- 0 replies
- 0 kudos
- 2140 Views
- 2 replies
- 2 kudos
SQL UDFs for DLT pipelines
I am trying to implement a UDF for a DLT pipeline. I have seen the documentation stating that it is possible but I am getting an error after adding an SQL UDF to a cell in the notebook attached to the pipeline. The aim is to have the UDF in a separat...
- 2140 Views
- 2 replies
- 2 kudos
- 2 kudos
You can't. The SQL support on DLT pipeline cluster is limited compared to a normal notebook. You can still define a UDF in Python using, of course, a Python notebook. In this case, you can use the spark.sql() function to execute your original SQL cod...
- 2 kudos
| User | Count |
|---|---|
| 1629 | |
| 790 | |
| 511 | |
| 349 | |
| 287 |
