- 1720 Views
- 5 replies
- 1 kudos
I have completed Data Engineer Associate (Version 3) Certificate. when we will get the Certificate in default?
I have completed Data Engineer Associate (Version 3) Certificate.when we will get the Certificate in default?
- 1720 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi @sherbin w Just a friendly follow-up, Are you still looking for help regarding this issue or you were able to received your voucher? please let us know
- 1 kudos
- 3025 Views
- 4 replies
- 2 kudos
How can I store timestamps using the local (not offset) timestamp instead of storing as UTC-Offset?
I have data that looks like this:2021-11-25T19:00:00.000-0500 2021-03-03T13:00:00.000-0500 2021-03-09T15:00:00.000-0500 2021-03-13T16:00:00.000-0500 2021-03-19T03:00:00.000-0400 2021-05-28T03:00:00.000-0400which is accurate, except I'm pulling the da...
- 3025 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @Jonathan Dufault Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us...
- 2 kudos
- 875 Views
- 1 replies
- 0 kudos
Are training/ecommerce data tables available as CSVs?
The course "Apache Spark™ Programming with Databricks" requires data sources such as training/ecommerce/events/events.parquet. Are these available as CSV files? My company's databricks configuration does not allow me to mount to such repositories, bu...
- 875 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Tim Tremper, The specific dataset you mentioned, "training/ecommerce/events/events.parquet", is in Parquet format, but you can easily convert it into a CSV format using Apache Spark™ on Databricks.Here's a step-by-step guide to convert the Parqu...
- 0 kudos
- 4361 Views
- 7 replies
- 0 kudos
My 'Data' menu item shows 'No Options' for Databases. How can I fix?
Hi, I'm new to Databricks and I've signed up for the Community edition.First, I've noticed that I cannot return to a previously created cluster, as I get the message telling me that restarting a cluster is not available to me. Ok, inconvenient, but I...
- 4361 Views
- 7 replies
- 0 kudos
- 0 kudos
Hi @Suteja Kanuri ,I get the error message under Data before I've created a cluster. Then I still get it when I've created a cluster and a notebook (having attached the notebook to the cluster). Thanks.
- 0 kudos
- 13209 Views
- 1 replies
- 0 kudos
How to parallelly merge data into partitions of databricks delta table using PySpark/Spark streaming?
I have a PySpark streaming pipeline which reads data from a Kafka topic, data undergoes thru various transformations and finally gets merged into a databricks delta table. In the beginning we were loading data into the delta table by using the merge ...
- 13209 Views
- 1 replies
- 0 kudos
- 0 kudos
@bobbysidhartha :When merging data into a partitioned Delta table in parallel, it is important to ensure that each job only accesses and modifies the files in its own partition to avoid concurrency issues. One way to achieve this is to use partition...
- 0 kudos
- 1719 Views
- 5 replies
- 0 kudos
FDBAv2 certificate
Hi Team I haven't received my data bricks vouchers as i have already submitted the Fundamentals of the databricks lakehouse platform V2 . Name: Pankaj Kumar Goyalemail: pankajaggarwal98@gmail.com
- 1719 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @Pankaj kumar Goyal Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best an...
- 0 kudos
- 1125 Views
- 2 replies
- 0 kudos
Visulization only from sample of data
When I display dataframe and add visualization, I can see a preview from only a sample of data, and when I confirm it, it is counted from all of the data. Until now, everything is fine. However, when I change the dataframe, the visualization is incon...
- 1125 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Ondrej Lostak Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so...
- 0 kudos
- 2112 Views
- 3 replies
- 1 kudos
Outdated - Databricks Data Engineer associate v2 lesson DE 4.2
While following the video lesson and executing the notebook 4.2, I noticed that creating the CREATE Table "users_jdbc" command generates an EXTERNAL table, while the video and, notebook too, suggests it as being a Managed table.Here are some printscr...
- 2112 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Tiago Barata Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...
- 1 kudos
- 8102 Views
- 6 replies
- 4 kudos
Need a way to show the non-trimmed data while query a table ?
When querying a json data , the values are getting trimmed. I need to see the full data for that field, is there any way to do so ?
- 8102 Views
- 6 replies
- 4 kudos
- 4 kudos
Hi @Raj Sethi We haven't heard from you since the last response from @Lakshay Goel and @Vigneshraja Palaniraj , and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, a...
- 4 kudos
- 2662 Views
- 7 replies
- 2 kudos
Resolved! Passed data engineer Associate exam , but didnt recive any badge / certificate .Please help
Passed data engineer Associate exam , but didnt recive any badge / certificate .Please help
- 2662 Views
- 7 replies
- 2 kudos
- 2 kudos
Hi, I gave the associate data engineer exam on 17 march, but I haven't received the certification.I got an email right after passing the certification that you would receive your certificate after 48 hours.Would you please look into my issue, thanks!...
- 2 kudos
- 1475 Views
- 3 replies
- 4 kudos
Efficiently orchestrate data bricks jobs
Hi Team,How efficiently can orchestrate data bricks jobs which involve a lot of transformations, dependencies, and complexity?At source have a lot of SSIS packages that have complex dependencies and more transformation. We have the following opti...
- 1475 Views
- 3 replies
- 4 kudos
- 4 kudos
My question is, how do we reliably orchestrate multiple Databricks Jobs/Workflows that are running in a mixed latency and can write to the same silver and gold delta tables? Could you please suggest the best approach and practices for the same?
- 4 kudos
- 1173 Views
- 2 replies
- 0 kudos
Real time data quality validation (Streaming data ingestion)
I was wondering how the Unity Catalog would help in data quality validations for real time (streaming data) data ingestion.
- 1173 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @arun laksh Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 0 kudos
- 5054 Views
- 2 replies
- 2 kudos
Resolved! Connect GraphQL to Data Bricks
Hi I am new to Databricks however I need to expose data found in the delta lake directly to GraphQL to be queried by several applications. Is there a connector or something similar to GraphQL that works with Databricks?
- 5054 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Andrew Camilleri Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feed...
- 2 kudos
- 1587 Views
- 2 replies
- 1 kudos
Resolved! Data Engineering with Databricks V2 not available in Partner
I cannot access the course anymore, it's shows it's under maintenance. For how long this will be? Is there any way to access it's videos for the Data Engineer Associate part?
- 1587 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Juan Manuel Moviglia Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tel...
- 1 kudos
- 2789 Views
- 7 replies
- 0 kudos
Resolved! Fetch new data from kinesis for every minute.
I want to fetch new data from kinesis source for every minute. I'm using "minFetchPeriod" option and specified 60s. But this doesn't seem to be working.Streaming query: spark \ .readStream \ .format("kinesis") \ .option("streamName", kinesis_stream_...
- 2789 Views
- 7 replies
- 0 kudos
- 0 kudos
Hi @Pranathi Girish Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedb...
- 0 kudos