- 2789 Views
- 2 replies
- 2 kudos
Badge not received for Databricks Lakehouse Fundamentals Accreditation
I have successfully passed the test. But I have'nt recieved any badge or points, could you help me with this?The e-mail registered in the community is the same as that registered in the databricks academy and in credentials.databricks.com as well.Pdf...
- 2789 Views
- 2 replies
- 2 kudos
- 2 kudos
Please submit a ticket to Databricks Training Team here: https://help.databricks.com/s/contact-us?ReqType=training ,they will reach out you soon, maybe due to holiday they revert late,please be patient
- 2 kudos
- 26411 Views
- 10 replies
- 6 kudos
Resolved! Permanently add python file path to sys.path in Databricks
If your notebook is in different directory or subdirectory than python module, you cannot import it until you add it to the Python path.That means that even though all users are using the same module, but since they are all working from different rep...
- 26411 Views
- 10 replies
- 6 kudos
- 6 kudos
setting the `spark.executorEnv.PYTHONPATH` did not work for me. it looked like Spark/Databricks overwrite this somewhere. I used a simple python UDF to print some properties like `sys.path` and `os.environ` and didn't see the path I added.Finally, I ...
- 6 kudos
- 3831 Views
- 2 replies
- 4 kudos
Databricks Terraform - how to manage databricks entirely through Terraform?
I'm stuck at a point where I can't automatically set up everything about a databricks environment due to the fact that service principals can't be made an admin at the account level (accounts.azuredatabricks.net, similar for aws). Going into a bare t...
- 3831 Views
- 2 replies
- 4 kudos
- 4 kudos
Unfortunately there are still some limitations with doing IaC on Databricks with Terraform (ex. another one is that you can't setup KeyVault as a secret store with Service Principal).I think that instead of doing stuff manually, you can authenticate ...
- 4 kudos
- 2772 Views
- 2 replies
- 2 kudos
Map("skipRows", "1") ignored during autoloader process. Something wrong with the format?
I've tried multiple variations of the following code. It seems like the map parameters are being completely ignored. CREATE LIVE TABLE a_raw2 TBLPROPERTIES ("quality" = "bronze") AS SELECT * FROM cloud_files("dbfs:/mnt/c-raw/a/c_medcheck_export*.csv"...
- 2772 Views
- 2 replies
- 2 kudos
- 3321 Views
- 2 replies
- 2 kudos
"org.apache.hadoop.hive.ql.metadata.HiveException: at least one column must be specified for the table" non deterministic error in a `insert ... select ... ` clause
Hi, first of all thahks for your work in databricks sql.Unfortunately i am having a problem running insert-selects statements programatically using the jdbc driver.They all have the form:`insert into `mytable` select 1, 'foo', moreLiterals`The statem...
- 3321 Views
- 2 replies
- 2 kudos
- 2 kudos
thanks for the suggestion, could tell me more about how to check logs in the cluster?
- 2 kudos
- 2511 Views
- 2 replies
- 2 kudos
Scalable MLwith Apache Spark course, introductory video "Install the courseware": "Repos" section missing on Databricks platform
Hello, I'm a beginner on Databricks. I have "community edition" account on Databricks platform and Partner account in Databricks Academy platform . The problem is that I don't have "Repos" section which I'm supposed to have, as it's said in the free ...
- 2511 Views
- 2 replies
- 2 kudos
- 2 kudos
Ok, thank you! What type of account is required to get access to "repos" for training purposes? Is it possible with any free account, or which one is necessary?
- 2 kudos
- 9914 Views
- 3 replies
- 7 kudos
Resolved! What exactly is Z Ordering and Bloom Filter?
Have gone through the documentation, still cannot understand it.How is bloom filter indexing a column different from z ordering a column?Can somebody explain to me what exactly happens while these two techniques are applied?
- 9914 Views
- 3 replies
- 7 kudos
- 7 kudos
hey @Daniel Sahal​ 1-A Bloomfilter index is a space-efficient data structure that enables data skipping on chosen columns, particularly for fields containing arbitrary textrefer this code snipet to create bloom filter CREATE BLOOMFILTER INDEX ON [TAB...
- 7 kudos
- 7647 Views
- 2 replies
- 2 kudos
Resolved! Allow read access to S3 buckets from one AWS accounts to other AWS accounts.
Dear team, We have several AWS accounts with S3 buckets, the databricks setup is on our dev AWS account and we would like to allow instance profile to have read permission on all our S3 buckets on the other AWS accounts ( without using bucket policy...
- 7647 Views
- 2 replies
- 2 kudos
- 2 kudos
Can you please share the IAM role policy in the secondary account (Bucket account) ?Just wanted to know have you tried setting the config in the cluster.spark.hadoop.fs.s3a.bucket.<s3-bucket-name>.aws.credentials.provider org.apache.hadoop.fs.s3a.aut...
- 2 kudos
- 2161 Views
- 2 replies
- 3 kudos
Advice on efficiently cleansing and transforming delta table
I have a delta table that is being updated nightly using Auto Loader. After the merge, the job kicks off a second notebook to clean and rewrite certain value using a series of UPDATE statements, e.g.,UPDATE TABLE foo SET field1 = some_value WHER...
- 2161 Views
- 2 replies
- 3 kudos
- 3 kudos
I would partition the table by some sort of date that autoloader can use. You could then filter your update further and it'll automatically use partition pruning and only scan related files.
- 3 kudos
- 2453 Views
- 1 replies
- 3 kudos
How do I programmatically get the database name in a DLT notebook?
I have configured a database in the settings of my DLT pipeline. Is there a way to retrieve that value programmatically from within a notebook? I want to do something likespark.read.table(f"{database}.table")
- 2453 Views
- 1 replies
- 3 kudos
- 3 kudos
You could also set it as a config value as database:value, and then retrieve it in the notebook using spark.conf.get().I'm hoping they update DLT to support UC, and then allow us to set database/schema at the notebook level in @dlt.table(schema_name,...
- 3 kudos
- 2263 Views
- 1 replies
- 3 kudos
Why does DLT CDC some time manifests the results table as a table and other times as a view?
I have a simple DLT pipeline that reads from an existing table, do some transformations, saves to a view, and then uses dlt.apply_changes() to insert the view into a results table. My question is:why is my results table a view and not a table like I ...
- 2263 Views
- 1 replies
- 3 kudos
- 3 kudos
I find most of my apply_changes tables are being created as materialized views as well. They do recalculate at runtime, so they're up to date and behave a lot like a table, but they aren't tables in the same sense.
- 3 kudos
- 5761 Views
- 2 replies
- 3 kudos
Giving new user workspace access
Hey all,We have a new user we'd like to give access to our spark workspace. We invited the user to the workspace as an account admin. They click on the invite link and create a password and login. Once logged in they can see the workspace and can ...
- 5761 Views
- 2 replies
- 3 kudos
- 3 kudos
The new feature in data bricks is identity federation if identity federation is enabled then the users part of the data bricks account and the account admin can assign the users to the workspace. The account admins can add the users from account cons...
- 3 kudos
- 3988 Views
- 3 replies
- 6 kudos
Resolved! Certification Spark - Can I please get an update on when should I receive the certificate?
- 3988 Views
- 3 replies
- 6 kudos
- 6 kudos
Hi @Mônica Borges Silva​ Thank you for reaching out! Please submit a ticket to our Training Team here: https://help.databricks.com/s/contact-us?ReqType=training and our team will get back to you shortly.
- 6 kudos
- 3524 Views
- 3 replies
- 2 kudos
Why there's no "New Union" option with Databricks connection
I'm trying to use databricks connect with tableau but it doesn't give me the "New Union" option. Is this normal or it's particular to me? My tableau desktop version is 2021.3
- 3524 Views
- 3 replies
- 2 kudos
- 2 kudos
there is option for connecting tableau find there SPARK SQL then it should work after adding proper connection string
- 2 kudos
- 39543 Views
- 13 replies
- 13 kudos
Certified Data Engineer Associate - v2 vs. v3 (Databricks Academy)
Which version of the Data Engineering with Databricks learning plan should I do? v2 or v3? Is there a Certified Data Engineer Associate V3 Exam already?Where can I find practice exams for Certified Data Engineer Associate V3?
- 39543 Views
- 13 replies
- 13 kudos
- 13 kudos
I would suggest choose v3 - it was latest version and covered more topic.
- 13 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
