Hello,I am trying to run the Classroom-Setup from the course files notebook (ASP 1.1 - Databricks Platform)(Course:Apache Spark™ Programming with DatabricksID: E-P0W7ZV)Instructions: "Setup:Run classroom setup to mount Databricks training datasets an...
In our azure pipeline we are using databricks-cli command to upload jar files at dbfs:/FileStore location and that works perfectly fine. But when we try to use the same command to upload files at dbfs:/Volume/dev/default/files, it does not work and g...
@Hardy I think you are using the word volume in the path but it should be Volumes(plural), not Volume(singular). Do one thing, copy the volume path directly from the Workspace and try.
Hello Databricks Community,in an IOT context we plan to ingest a large amount of JSON files (~2 Million per Day). The JSON files are in json lines format und need to be compressed on the IOT devices. We can provide suggestions for the type of compres...
Hi, sorry I guess my response wasn't sent. The source are JSON files that are uploaded to an S3 bucket. The sink will be a Delta Table and we are using autoloader.The question was about the compression format of the incoming JSON files, e.g. if it wo...
Does anyone know what the first digits of the jobgroup that are shown in the spark ui mean when using all purpose clusters to launch multiple jobs?Right now the pattern is something like: [id_random]_job_[jod_id]_run-[run_id]_action_[action].
Hi @FerArribas The first digits of the jobgroup that are shown in the spark UI are execContextId and cmdId(Command_ID). You can think of the execContextId as some kind of “REPL ID” For example, if you take the below job group ID as an example, jobGr...
Hi !! I have a question is it possible to implementing liquid clustering for DataFrames directly saved to delta files (df.write.format("delta").save("path")), The conventional approach involving table creation
Hi,Hopefully this question is related to testing and any production data would get persisted to a table but one example is:df = (spark.range(10).write.format("delta").mode("append").save("file:/tmp/data"))ALTER TABLE delta.`file:/tmp/data` CLUSTER BY...
Hi,I ran my code sucessfully in the past but suddenly it stopped working. I have a python code that transfer local files to DBFS location using CLI. When I run the command manually on the screen, it works but in the code, it gives me the error "retu...
The 127 error code indicates “command not found”,Try using the full path of the databricks command
Hello Esteemed Community,I have a fundamental question to ask, and I approach it with a sense of humility. Your guidance in my learning journey would be greatly appreciated. I am eager to learn how to build a hands-on data pipeline within the Databri...
Hey @kazinahian I completely understand your hesitation and appreciate your approach to seeking guidance! Embarking on a learning journey can be daunting, especially when financial considerations are involved. I'm happy to offer some advice on buildi...
I am getting an error every time I try to view another branch or create a branch. Sometimes this has happened in the past, but usually seems to fix itself after about 10-30 minutes. This error has been lasting for over 12 hours, so I am now concerned...
I had the same problem and I could resolve it by creating the repo again with a trailing ".git" in the Git repository URL.For example, use thishttps://gitlab.mycompany.com/my-project/my-repo.gitnot this:https://gitlab.mycompany.com/my-project/my-repo...
Currently, I am both an account administrator and also a workspace administrator in Databricks.When I try to enable the entitlements "Workspace access" and "Databricks SQL access" to account groups I am receiving the error "Failed to enable entitlem...
Hi @Arnold_Souza, The error "Failed to enable entitlement.: Group not found" that you're experiencing when trying to enable the entitlements “Workspace access” and “Databricks SQL access” for account groups is likely due to the fact that Identity Fed...
I just opened another issue: https://issues.apache.org/jira/browse/SPARK-46959It corrupts data even when read with mode="FAILFAST", i consider it critical, because basic stuff like this should just work!
either: [ 'some text', 'some text"', 'some text"' ]alternatively: [ '"some text"', 'some text"', 'some text"' ]probably most sane behavior would be a parser error ( with mode="FAILFAST").just parsing garbage without warning the user is certainly not...
Hello,we are starting to experiment with Databricks Asset Bundles, especially to keep jobs aligned between workspaces. Is there a way to start managing existing jobs, to avoid erasing previous runs history?Thank you,Tommaso
Hi there! I'm pretty new to using Auto Loader, so this may be a really obvious fix, but it's stumped me for a few weeks, so I'm hoping someone can help! I have a small csv file saved in ADLS with a list of pizzas for an imaginary pizza restaurant. I'...
So, I figured out what the issue was. I needed to delete checkpoint folder. After I did this and re-ran the notebook, everything worked fine!
HelloWe have a workflow in our team to perform usual monthly tasks to be ran on the first working day of the month.Each of the ~20 users will run a clone of this workflow most likely all around the same time but with different options. Because we don...
maybe you can set a lock before call SOAP APIpython - Using a Lock with redis-py - Stack Overflow
How do i create an array from a field while applying sorting?how do I do this in data brick since databricks does not support order by in array_agg? same is possible in Snowflake and(Array agg) or Redshift(listagg). SELECT ARRAY_AGG(O_ORDERKEY) WITH...
%sql SELECT array_sort(array_agg(col) ,(left, right) -> CASE WHEN left < right THEN -1 WHEN left > right THEN 1 ELSE 0 END) arr_col FROM VALUES (3), (2), (1) AS tab(col); https://docs.databricks.com/en/sql/language-manual/functions/array_sort.h...
I have a Python Databricks notebook which I want to call/run another Databricks notebook using dbutils.notebook.run()... but I want to run all the cells in the "called" notebook except the last one.Is there a way to do a count of cells in the called ...
In the alternative way, you can use dbutils.notebook.run to pass the parameters, and use dbutils.widgets.get in another notebook to get the parameter values,and determine the parameter values to decide whether to execute codes in the specified cellh...
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 566 | |
| 349 | |
| 287 |
