- 8686 Views
- 3 replies
- 6 kudos
Question on Databricks Spark online proctored exam
Some silly questions folks. I took online proctored Databricks spark certification couple of days back and my unofficial result was pass. I received a mail that it might https://speedtest.vet/ take upto one week to receive the certification, if awar...
- 8686 Views
- 3 replies
- 6 kudos
- 6 kudos
better would have been to ask for permission before drinking. I can share my exp. My mobile alarm started buzzing during the exam, I requested the moderator, he then paused the exam and asked me to take my laptop to the mobile and then to switch off,...
- 6 kudos
- 17613 Views
- 7 replies
- 13 kudos
How to read file in pyspark with “]|[” delimiter
The data looks like this: pageId]|[page]|[Position]|[sysId]|[carId 0005]|[bmw]|[south]|[AD6]|[OP4 There are atleast 50 columns and millions of rows. I did try to use below code to read: dff = sqlContext.read.format("com.databricks.spark.csv").option...
- 17613 Views
- 7 replies
- 13 kudos
- 13 kudos
you might also try the blow option.1). Use a different file format: You can try using a different file format that supports multi-character delimiters, such as text JSON.2). Use a custom Row class: You can write a custom Row class to parse the multi-...
- 13 kudos
- 32622 Views
- 4 replies
- 3 kudos
Resolved! Set environment variables in global init scripts
Hi Databricks Community,I want to set environment variables for all clusters in my workspace.The goal is to have environment (dev, prod) specific environment variables values.Instead of set the environment variables for each cluster, a global script ...
- 32622 Views
- 4 replies
- 3 kudos
- 3 kudos
We have set the env variable at Global Init script as below,sudo echo DATAENV=DEV >> /etc/environmentand we try to access the variable in notebook that run with "Shared" cluster mode. import os print(os.getenv("DATAENV"))But the env variable is not a...
- 3 kudos
- 4065 Views
- 3 replies
- 3 kudos
How to publish Notebook Dashboard without possiblity to "exit" FullScreen?
Is there a way to remove the "exit" Button from the fullscreen within the sparks Notebook - Dashboard ?
- 4065 Views
- 3 replies
- 3 kudos
- 3 kudos
Could you please share a screenshot of what you see. I dont see any exit button. Or I might be looking at a wrong place.
- 3 kudos
- 37846 Views
- 15 replies
- 2 kudos
Resolved! How to create connection between Databricks & BigQuery
Hi, I would like to connect our BigQuery env to Databricks, So I created a service account but where should I configure the service account in Databricks? I read databricks documention and it`s not clear at all. Thanks for your help
- 37846 Views
- 15 replies
- 2 kudos
- 2 kudos
@kfiry adding to @Werner Stinckens did you added projectid in read spark query , projectid should be one where big query instance running. also please follow best practices in terms of egress data cost spark.read.format("bigquery") \ .option("tabl...
- 2 kudos
- 9128 Views
- 2 replies
- 2 kudos
Resolved! What is the best way to find deltalake version on OSS and Databricks at runtime?
To identify certain deltalake features available on a certain installation, it is important to have a robust way to identify deltalake version. For OSS, I found that the below Scala snippet will do the job.import io.delta println(io.delta.VERSION)Not...
- 9128 Views
- 2 replies
- 2 kudos
- 2 kudos
@Yousry Mohamed - could you please check the DBR runtime release notes for the Delta lake API compatibility matrix section ( DBR version vs Delta lake compatible version) for the mapping.Reference: https://docs.databricks.com/release-notes/runtime/r...
- 2 kudos
- 4217 Views
- 3 replies
- 2 kudos
Resolved! Limitation as of now in delta live table
I am thinking of using delta live table, before that I want to be aware of the limitations it has as of now when it s announced on datasummit 2021
- 4217 Views
- 3 replies
- 2 kudos
- 2 kudos
There doesn't appear to be a way to enforce a retention policy on source tables when defining a structured stream. Setting the options for "ignoreChanges" and "ignoreDeletes" doesn't seem to have any effect at all. CDC does not fill this role either,...
- 2 kudos
- 5298 Views
- 2 replies
- 0 kudos
GRANT PRIVILEGES or REVOKE not working in databricks: Operation not allowed
Hi everyone!I am having a problem! I can Grant or Revoke privileges from users using the UI on databricks, but when I try to do that using SQL commands such as GRANT SELECT ON SCHEMA [... ] TO [USER]; I get an error stating Operation not allowed.I am...
- 5298 Views
- 2 replies
- 0 kudos
- 2690 Views
- 2 replies
- 0 kudos
Issue with visualizing dataframe from a job
Greetings,I have the following data set:```sqlSELECT * FROM ( VALUES ('2023-02',113.81::decimal(27,2),'A','X'), ('2023-02',112.66::decimal(27,2),'A','Y'), ('2023-02',1223.8::decimal(27,2),'B','X'), ('2023-02',1234.56::decimal(27,2),'B',...
- 2690 Views
- 2 replies
- 0 kudos
- 0 kudos
Attaching some more screenshots to add more details.This seems to be a bug in the bar chart visualization widget when displaying from job run.
- 0 kudos
- 2139 Views
- 1 replies
- 3 kudos
Updated UI for Create, run, and manage Databricks Jobs
Updated UI for Create, run, and manage Databricks Jobs
- 2139 Views
- 1 replies
- 3 kudos
- 3 kudos
Hi Ajay, Could you please clarify the ask here? Thanks.
- 3 kudos
- 2020 Views
- 1 replies
- 4 kudos
Resolved! REVOKE access from users
There is a use-case where we want to REVOKE access from users so that they can't run VACUUM Command on Delta Table.Can anyone please help here.
- 2020 Views
- 1 replies
- 4 kudos
- 4 kudos
Hello @KVNARK . We cannot specifically restrict Vacuum operation alone.You need to remove "MODIFY" access on the table and restrict only to the "Read" (SELECT) operationPlease note if you restrict to only "read" it will also affect all the write, up...
- 4 kudos
- 2122 Views
- 1 replies
- 1 kudos
Using your own docker container to launch databricks cluster.
When using your own docker container while creating a databricks cluster, what is the mapping between the number of containers launched and the nodes launched?Is it 1:1 mapping? or is it similar to other orchestration framework like Kubernetes?Or is ...
- 2122 Views
- 1 replies
- 1 kudos
- 1 kudos
+1
- 1 kudos
- 4953 Views
- 4 replies
- 5 kudos
Data Engineer Associate and Professional tittle holders count
How many people hold tittles of certified Databricks Data Engineers Asociate/Professional right now?Is there any place I can check the global certificate count?
- 4953 Views
- 4 replies
- 5 kudos
- 5 kudos
check here: https://credentials.databricks.com/collection/da21363e-5c7d-410a-b144-dd07d3e22942?_ga=2.163643839.1823848454.1674389186-2106443313.1667211405&_gac=1.49521364.1672812437.CjwKCAiAwc-dBhA7EiwAxPRylBN9S-JeQ8779ec3GXJYBQPfnu_qkv5l_MKO1u4jw2w-...
- 5 kudos
- 4625 Views
- 3 replies
- 1 kudos
Resolved! Azure CosmosDB change feed ingestion via DLT
Is there a way to ingest Azure CosmosDB data via Delta Live Tables? If I use regular workflows it works well, but with DLT I'm not able to set CosmosDB Connector on a cluster.
- 4625 Views
- 3 replies
- 1 kudos
- 1 kudos
Thanks a lot! Just wanted to doublecheck whether this natively exists.
- 1 kudos
- 3772 Views
- 2 replies
- 0 kudos
depth of view exceeds the maximum view resolution depth (100).
I got this error after updating a view. How can I increase the value of spark.sql.view.maNestedViewDepth to work around this? Thanks!
- 3772 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi, Could you please confirm if you are showing the view? (https://docs.databricks.com/sql/language-manual/sql-ref-syntax-aux-show-views.html) also, it will be helpful if you post the screenshot of the error.
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 555 | |
| 349 | |
| 287 |
