- 8377 Views
- 4 replies
- 1 kudos
How to merge two separate DELTA LIVE TABLE?
So I have two delta live tables. One that is the master table that contains all the prior data, and another table that contains all the new data for that specific day. I want to be able to merge those two table so that the master table contains would...
- 8377 Views
- 4 replies
- 1 kudos
- 958 Views
- 0 replies
- 1 kudos
While accessing the data on recipient side using delta_sharing.load_table_changes_as_spark(), it shows data of all versions.
When I tried to access specific version data and set the arguments value to the specific number, I get all version data.data1 = delta_sharing.load_table_changes_as_spark(table_url, starting_version=1, ending_version=1)data2 = delta_sharing.load_table...
- 958 Views
- 0 replies
- 1 kudos
- 1232 Views
- 0 replies
- 1 kudos
Trouble Displaying Full Size Images from Spark Dataframe
Hi, I have followed this guide (https://learn.microsoft.com/en-us/azure/databricks/_static/notebooks/image-data-source.html) to successfully load some image data into a spark df and display it as a thumbnail. I would like to display a single image fr...
- 1232 Views
- 0 replies
- 1 kudos
- 3998 Views
- 3 replies
- 6 kudos
Resolved! Function When + Dictionary.
Hey everyone, I'm avoiding repeating the When Function for 12x, so I thought of the dictionary. I don't know if it's a limitation of the Spark function or a Logic error. Does the function allow this concatenation?
- 3998 Views
- 3 replies
- 6 kudos
- 6 kudos
Hello everyone, I found this alternative to reduce repeated code.custoDF = (custoDF.withColumn('month', col('Nummes').cast('string')) .replace(months, subset=['month']))
- 6 kudos
- 2782 Views
- 3 replies
- 0 kudos
HTTP 403 on git-credentials API
Hi,I am trying to set git credentials for my service principal. I follow the process described here but I get a 403 error when making the POST request to ${DATABRICKS_HOST}/api/2.0/git-credentials with service principal token.By the way, I also canno...
- 2782 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Sébastien FALQUIER it works for me, there are no restrictions. Maybe the PAT token you generated for the service principle got expired. Can you generate a new token and try to run GET/git-credentials API?How are you creating PAT for service prin...
- 0 kudos
- 3492 Views
- 4 replies
- 2 kudos
Cloud provider launch failure
When I want to create a cluster a get this error message:DetailsAWS API error code: InvalidGroup.NotFoundAWS error message: The security group 'sg-0ded75eefd66bf421' does not exist in VPC 'vpc-0ec7da3d5977f6ec9'And when I inspect the security groups ...
- 3492 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi, I am running into a similar issue. but in my case, the security has been deleted by mistake.Is there a way to make Databricks recreate the missing group ?@Kaniz Fatma , where can the CreateSecurityGroup command be ran ? Does it change the securi...
- 2 kudos
- 832 Views
- 0 replies
- 0 kudos
Structured Streaming
I need some solution for below problem.We have set of json files which are keep coming to aws s3, these files contains details for a property . please note 1 property can have 10-12 rows in this json file. Attached is sample json file.We need to read...
- 832 Views
- 0 replies
- 0 kudos
- 3940 Views
- 4 replies
- 13 kudos
Resolved! To practice Databricks SQL
Is there any sand box kind of thing where we can do some hands-on on Databricks SQL/run the Note books attaching to the Clusters apart from the free trial provided by Databricks.
- 3940 Views
- 4 replies
- 13 kudos
- 13 kudos
Databricks SQL workspace will be available only for Databricks Premium service. If you have Azure Pass subscription, then you can able to get it for practicing it.
- 13 kudos
- 13097 Views
- 3 replies
- 3 kudos
Resolved! UUID/GUID Datatype in Databricks SQL
Hi all,I am trying to create a table with a GUID column.I have tried using GUID, UUID; but both of them are not working.Can someone help me with the syntax for adding a GUID column?Thanks!
- 13097 Views
- 3 replies
- 3 kudos
- 3 kudos
Hey @Avinash Narasimhan , What is the exact problem you are getting can you please share it is working fine for meThanksAviral Bhardwaj
- 3 kudos
- 1842 Views
- 4 replies
- 0 kudos
Hi, Has anyone cleared professional DE? please advise on professional data engineer exam. will advance DE learning path be sufficient? Or need to fol...
Hi,Has anyone cleared professional DE? please advise on professional data engineer exam. will advance DE learning path be sufficient? Or need to follow some other resource as well.
- 1842 Views
- 4 replies
- 0 kudos
- 0 kudos
Hello Have a look at this link http://msdatalab.net/how-to-pass-the-professional-databricks-data-engineering/
- 0 kudos
- 1854 Views
- 1 replies
- 0 kudos
Does Databricks provide a mechanism to have rate limiting for receivers?
from pyspark.sql import SparkSession scala_version = '2.12' spark_version = '3.3.0' packages = [ f'org.apache.spark:spark-sql-kafka-0-10_{scala_version}:{spark_version}', 'org.apache.kafka:kafka-clients:3.2.1' ] spark = SparkSession.bui...
- 1854 Views
- 1 replies
- 0 kudos
- 0 kudos
hi @Software Architect i dont think so
- 0 kudos
- 3669 Views
- 7 replies
- 1 kudos
Resolved! Badge Not Received for - Databricks Lakehouse Fundamentals Accreditation (V2)
Hi TeamI have passed the Databricks Lakehouse Fundamentals Accreditation (V2) on Dec 8th.Still have not received the Badge in credentials or any email of that kind.Please have a look.@Kaniz Fatma
- 3669 Views
- 7 replies
- 1 kudos
- 1 kudos
I wonder how much attempt you set to create this type of excellent informative web site.marco's pizza starr avedomino's pizza price
- 1 kudos
- 2448 Views
- 3 replies
- 2 kudos
December exam free voucher for Databricks Certified Associate Developer for Apache Spark 3.0 exam.
Dear @Vidula Khanna Hope you're having great day. This is of HIGH priority for me, I've to schedule exam in December before slots are full.I gave Databricks Certified Associate Developer for Apache Spark 3.0 exam on 30th Nov but missed by one perc...
- 2448 Views
- 3 replies
- 2 kudos
- 2 kudos
hey @Smitha Nelapati ,you can attend the below webinars and get the 75% off in Jan
- 2 kudos
- 3398 Views
- 3 replies
- 6 kudos
Unable to signup to a Databricks community edition account
As you can see, I get the error underneath the big orange button. This is after I click the link at the bottom to try the community edition. I have tried a couple of locations since I am currently based in South Korea but I am actually from the UK. T...
- 3398 Views
- 3 replies
- 6 kudos
- 6 kudos
Hi @Muhammad Ali Just a friendly follow-up. Are you able to log in to your Community-Edition account? If yes, then mark the answer as best or if you need further assistance kindly let me know. Thanks and Regards
- 6 kudos
- 3096 Views
- 1 replies
- 0 kudos
Write streaming output to DynamoDB
Hi All,I am trying to write a streaming DF into dynamoDB with below code.tumbling_df.writeStream \ .format("org.apache.spark.sql.execution.streaming.sinks.DynamoDBSinkProvider") \ .option("region", "eu-west-2") \ .option("tableName", "PythonForeac...
- 3096 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @SUDHANSHU RAJ ,I can't seem to find much on the "DynamoDBSinkProvider" source. Have you checked out the link for the streaming to DynamoDB documentation?
- 0 kudos
| User | Count |
|---|---|
| 1612 | |
| 768 | |
| 348 | |
| 286 | |
| 252 |
