Hi,I was checking the EC2 details in our AWS account, and found all the EC2's launched from Databricks are with "dedicated" tenancy. I double checked the cluster launch configuration and didn't find anywhere to change the tenancy setting. How can we ...
Connect with Databricks support they will guide in this
Spark Dataframes SchemaSchema inference is not reliable.We have the following problems in schema inference:Automatic inferring of schema is often incorrectInferring schema is additional work for Spark, and it takes some extra timeSchema inference is ...
one other difference between those 2 approaches is that In Schema DDL String approach we use STRING, INT etc.. But In Struct Type Object approach we can only use Spark datatypes such as StringType(), IntegerType(), etc..
df = spark.createDataFrame([(2018,'Apple1',45000),(2019,'Apple1',35000),(2020,'Apple1',75000), (2018,'Samsung',15000),(2019,'Samsung',20000),(2020,'Samsung',25000), (2018,'Nokia',21000),(2019,'Nokia',17000),(2020,'Nokia',140...
Issue is coming because of the semicolon beside PhoneBrandSales. Try removing that ; issue will be resolved. Please refer to the screenshot below.Please select this answer as best answer if it resolved your issueThanks,Varshith
Hi, I've registered some days ago to partner-academy.databricks using my company ID but that day I didn't login. Now, I'm trying to login but it's impossible, I'm also trying to recover my password through the forgot password option but never get the...
Hi @Cristhian Plazas​ Thanks for reaching us!Kindly, mail to Nadiya to resolve the issue.
Hi, I have successfully passed the test after completion of the course but I have'nt recieved any badge from your side. I simply got a certificate from your side post the test completion. I need to post my credentials on Linkedin with valid verificat...
Hi @Anirudh Negi​ Thank you for reaching out! Please submit a ticket to our Training Team here: https://help.databricks.com/s/contact-us?ReqType=training and our team will get back to you shortly.
I have successfully passed the test. But I have'nt recieved any badge or points, could you help me with this?The e-mail registered in the community is the same as that registered in the databricks academy and in credentials.databricks.com as well.Pdf...
Please submit a ticket to Databricks Training Team here: https://help.databricks.com/s/contact-us?ReqType=training ,they will reach out you soon, maybe due to holiday they revert late,please be patient
If your notebook is in different directory or subdirectory than python module, you cannot import it until you add it to the Python path.That means that even though all users are using the same module, but since they are all working from different rep...
setting the `spark.executorEnv.PYTHONPATH` did not work for me. it looked like Spark/Databricks overwrite this somewhere. I used a simple python UDF to print some properties like `sys.path` and `os.environ` and didn't see the path I added.Finally, I ...
I'm stuck at a point where I can't automatically set up everything about a databricks environment due to the fact that service principals can't be made an admin at the account level (accounts.azuredatabricks.net, similar for aws). Going into a bare t...
Unfortunately there are still some limitations with doing IaC on Databricks with Terraform (ex. another one is that you can't setup KeyVault as a secret store with Service Principal).I think that instead of doing stuff manually, you can authenticate ...
I've tried multiple variations of the following code. It seems like the map parameters are being completely ignored. CREATE LIVE TABLE a_raw2 TBLPROPERTIES ("quality" = "bronze") AS SELECT * FROM cloud_files("dbfs:/mnt/c-raw/a/c_medcheck_export*.csv"...
skipRows was added in DBR 11.1 -- what DBR is your DLT pipeline on?
Hi, first of all thahks for your work in databricks sql.Unfortunately i am having a problem running insert-selects statements programatically using the jdbc driver.They all have the form:`insert into `mytable` select 1, 'foo', moreLiterals`The statem...
thanks for the suggestion, could tell me more about how to check logs in the cluster?
Hello, I'm a beginner on Databricks. I have "community edition" account on Databricks platform and Partner account in Databricks Academy platform . The problem is that I don't have "Repos" section which I'm supposed to have, as it's said in the free ...
Ok, thank you! What type of account is required to get access to "repos" for training purposes? Is it possible with any free account, or which one is necessary?
Have gone through the documentation, still cannot understand it.How is bloom filter indexing a column different from z ordering a column?Can somebody explain to me what exactly happens while these two techniques are applied?
hey @Daniel Sahal​ 1-A Bloomfilter index is a space-efficient data structure that enables data skipping on chosen columns, particularly for fields containing arbitrary textrefer this code snipet to create bloom filter CREATE BLOOMFILTER INDEX ON [TAB...
Dear team, We have several AWS accounts with S3 buckets, the databricks setup is on our dev AWS account and we would like to allow instance profile to have read permission on all our S3 buckets on the other AWS accounts ( without using bucket policy...
Can you please share the IAM role policy in the secondary account (Bucket account) ?Just wanted to know have you tried setting the config in the cluster.spark.hadoop.fs.s3a.bucket.<s3-bucket-name>.aws.credentials.provider org.apache.hadoop.fs.s3a.aut...
I have a delta table that is being updated nightly using Auto Loader. After the merge, the job kicks off a second notebook to clean and rewrite certain value using a series of UPDATE statements, e.g.,UPDATE TABLE foo SET field1 = some_value WHER...
I would partition the table by some sort of date that autoloader can use. You could then filter your update further and it'll automatically use partition pruning and only scan related files.
I have configured a database in the settings of my DLT pipeline. Is there a way to retrieve that value programmatically from within a notebook? I want to do something likespark.read.table(f"{database}.table")
You could also set it as a config value as database:value, and then retrieve it in the notebook using spark.conf.get().I'm hoping they update DLT to support UC, and then allow us to set database/schema at the notebook level in @dlt.table(schema_name,...
| User | Count |
|---|---|
| 1602 | |
| 736 | |
| 344 | |
| 284 | |
| 247 |
