- 5966 Views
- 3 replies
- 4 kudos
Resolved! Spark persistent view on a partition parquet file
In Spark, is it possible to create a persistent view on a partitioned parquet file in Azure BLOB? The view must be available when the cluster restarted, without having to re-create that view, hence it cannot be a temp view.I can create a temp view, b...
- 5966 Views
- 3 replies
- 4 kudos
- 4 kudos
Here is what worked for me. Hope this helps someone else: https://stackoverflow.com/questions/72913913/spark-persistent-view-on-a-partition-parquet-file/72914245#72914245CREATE VIEW test as select * from parquet.`/mnt/folder-with-parquet-file(s)/`@Hu...
- 4 kudos
- 1222 Views
- 0 replies
- 0 kudos
BEGINNER
Hey i am beginner and i am intrested to learn data engineering path for azure databrics.Can some one please provide some guidance
- 1222 Views
- 0 replies
- 0 kudos
- 1790 Views
- 1 replies
- 2 kudos
How to get the normalized hours value for the Databricks EMR cluster?
Hi Hubert!I'm working on a usecase to get the compute usage stats. I used boto3 code and describe_cluster() function to get the normalized instance hours value for the EMR cluster. I would like to know if there is an equivalent for this normalized in...
- 1790 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi Bala, nice to meet you.Yes, there is an equivalent. For example, in Databricks, you need to use the get method in cluster API or CLI implementation of API.https://docs.databricks.com/dev-tools/api/latest/clusters.html#get
- 2 kudos
- 19533 Views
- 1 replies
- 2 kudos
Creating permanent views from dataframes?
Hi:It's possible to create temp views in pyspark using a dataframe (df.createOrReplaceTempView()), and it's possible to create a permanent view in Spark SQL. But as far as I can tell, there is no way to create a permanent view from a dataframe, somet...
- 19533 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi Kaniz:This is what I understood from the research I did, I was curious more as to why permanent views can't be created from dataframes and whether this is a feature that might be implemented by Databricks or Spark at some point. Temporary views ca...
- 2 kudos
- 1230 Views
- 0 replies
- 0 kudos
www.mca.gov.in
Today, the entire private limited company registration process and other regulatory filings are paperless; documents are filed electronically through the MCA website and is processed at the Central Registration Centre (CRC). Online Private Limited Co...
- 1230 Views
- 0 replies
- 0 kudos
- 2829 Views
- 2 replies
- 2 kudos
Simba ODBC datetime with millisecond overflows
Hello,Using odbc 2.6.24.1041-2 for Linux, when inserting rows with milliseconds precision date (e.g. 2022-07-03 13:57:48.500) precision I get:2022/07/03 14:41:19 SQLExecute: {22008} [Simba][Support] (40520) Datetime field overflow resulting from inva...
- 2829 Views
- 2 replies
- 2 kudos
- 2 kudos
I was passing a string (e.g. '2022-07-03 13:57:48.500') to the Golang SQL driver which is not working if the ms part is specified, but otherwise it works (e.g. '2022-07-03 13:57:48'). Passing a native Golang time.Time seems to work for timestamps wit...
- 2 kudos
- 1062 Views
- 0 replies
- 5 kudos
Databricks Office Hours Register for Office Hours to participate in a live Q&A session and receive technical support directly from Databricks expe...
Databricks Office HoursRegister for Office Hours to participate in a live Q&A session and receive technical support directly from Databricks experts! Our next events are scheduled for July 13th & July 27th from 8:00am - 9:00am PT | 3:00pm - 4:00pm GM...
- 1062 Views
- 0 replies
- 5 kudos
- 1345 Views
- 0 replies
- 0 kudos
Unexpected behaviour when creating multiple Account-Level Accounts
We've had a qustion regarding possibly unexpected behaviour when creating multiple accounts on the account-level on https://accounts.cloud.databricks.com/.Short Version:It's possible to create multiple accounts with different letter-cases on https://...
- 1345 Views
- 0 replies
- 0 kudos
- 3998 Views
- 3 replies
- 1 kudos
Resolved! ipywidgets not working in DBR 11.0 on Community Edition
I'm looking forward to using ipywidgets which should be working in DBR 11.0 as they provide more options when creating a notebook UI. I saw that DBR 11.0 is available as of yesterday so I created a test cluster in the Databricks Community Edition ju...
- 3998 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Wayne Deleersnyder​ I was able to import ipywidgets in DBR 11.0. As you can see in the output below. The slider is visibleYou are facing this issue probably because the community edition has limited access. To get all the features you should at l...
- 1 kudos
- 1481 Views
- 3 replies
- 2 kudos
pd.read_csv failed
https://i.imgur.com/PAGzSr9.png
- 1481 Views
- 3 replies
- 2 kudos
- 2 kudos
Thanks for your kind reply:Below works for me:https://imgur.com/BmMzatIBut why, as you mentioned, using the classic path, below does not work?https://imgur.com/Ba1a4Iv
- 2 kudos
- 1191 Views
- 0 replies
- 0 kudos
How to make a paper airplane
How to make a paper airplane
- 1191 Views
- 0 replies
- 0 kudos
- 1806 Views
- 2 replies
- 0 kudos
DBrick workspace URL block outside the corporate network?
As per security concern, need to restrict/block the dbricks workspace url outside the corporate network. Tried below ip access list, it able to restrict only user login access out the corporate network but still the workspace id url is live outside t...
- 1806 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @as999​ Hope everything is going great!Does @Atanu Sarkar​'s response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly? Else please let us know if you need more help. We'...
- 0 kudos
- 5020 Views
- 3 replies
- 1 kudos
How do you install R packages?
I've tried this, but it doesn't appear to be working: https://community.databricks.com/s/question/0D53f00001GHVX1CAP/unable-to-install-sf-and-rgeos-r-packages-on-the-clusterWhen I run the following after that init script, I receive an error.library(r...
- 5020 Views
- 3 replies
- 1 kudos
- 1 kudos
Hey there @Christopher Flach​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear fr...
- 1 kudos
- 5891 Views
- 5 replies
- 0 kudos
Resolved! A void column was created after connecting to cosmos
Hi everyone, I have connected to Cosmos using this tutorial https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/cosmos/azure-cosmos-spark_3_2-12/Samples/DatabricksLiveContainerMigrationAfter creating a table using a simple SQL command:CREATE TA...
- 5891 Views
- 5 replies
- 0 kudos
- 0 kudos
Hey there @Joel iemma​ Hope all is well! Just wanted to check in if you would be happy to mark an answer as best for us, please? It would be really helpful for the other members too.Cheers!
- 0 kudos
- 3652 Views
- 2 replies
- 0 kudos
Resolved! JDBC Connectivity via workspace url when No Public IP selected.
Hi All, I think I might be missing something in regard to No Pubic IP Clusters. I have set this option on a workspace (Azure) and setup the appropriate subnets. To my surprise, when I went to setup a JDBC connection to the cluster the JDBC connec...
- 3652 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey there @Ashley Betts​ Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best? It would be really helpful for the other members too.Cheers!
- 0 kudos
| User | Count |
|---|---|
| 1613 | |
| 771 | |
| 349 | |
| 286 | |
| 253 |
